feat: toHaveScreenshot drop-in — route existing Playwright assertions through Percy - #637
Closed
Shivanshu-07 wants to merge 7 commits into
Closed
feat: toHaveScreenshot drop-in — route existing Playwright assertions through Percy#637Shivanshu-07 wants to merge 7 commits into
Shivanshu-07 wants to merge 7 commits into
Conversation
…sertions through Percy
Adds @percy/playwright/dropin: one require line in playwright.config.js
overrides Playwright's toHaveScreenshot() so every existing screenshot
assertion is captured and uploaded to Percy, with the suite never failing on
a Percy problem (missing token / CLI down → the whole run transparently falls
back to native toHaveScreenshot).
Registration writes the matcher into Playwright expect's internal userMatchers
(call-time dispatch spreads them after built-ins) because the public
expect.extend() silently ignores built-in matcher names on the shared
instance; on an unrecognized expect shape it degrades to extend() WITH a loud
warning — never a silent no-op. Verified working on @playwright/test 1.60/1.61.
Capture modes (config file .percy-playwright-dropin.json):
- screenshot (default): the Playwright-stabilized PNG is uploaded as a raw
comparison (generic/app projects); snapshot identity (name/browser/width)
mirrors Playwright's on-disk baseline naming byte-for-byte, and tag
dimensions are parsed from the PNG (percy-api validates height).
- snapshot: serialized-DOM web snapshot for web projects — DELEGATES to this
repo's own captureDOM, so the drop-in inherits the readiness gate,
responsive capture and cross-origin iframe handling and cannot drift from
percySnapshot(). Locator subjects become element-scoped snapshots via a
transient data attribute + scope.
First-build-as-baseline: the bundled percy-playwright wrapper bin flags the
run's build as a baseline candidate (PERCY_DROPIN_BASELINE_CANDIDATE, mirrors
PERCY_BUILD_SOURCE); percy-api decides first-ness server-side, and on a
genuine first build the dropin globalSetup seeds the repo's committed
Playwright baseline PNGs as the build's content (8-way parallel) while the
override skips live captures — the server auto-approves it (flag-gated), so
diffs start on the very next run. Also ships the opt-in fail-on-changes gate
reporter and sync-assertion mode.
Compat fixes surfaced by the @playwright/test 1.24→1.60 devDep bump:
- ENV_INFO's top-level require('playwright/package.json') crashes at import
time under the 1.61 runner (CJS↔ESM interop) — now guarded, degrades to
@playwright/test's version or a bare label
- the dropin↔root require is lazy for the same interop reason
Tests: dispatch specs drive the REAL Playwright expect through the override
and hard-assert the posted comparison (retry-wrapped against the shared
testing-server reset race); unit specs pin name derivation, first-build
seeding and locator scoping. 117 passed, lint clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
The toHaveScreenshot drop-in needs modern @playwright/test (expect internals), whose engine floor is Node 18. Runtime support for the core SDK entry is unchanged; this bumps the dev/CI toolchain only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Playwright >=1.39 no longer downloads browsers as a package postinstall; the suite failed on CI with 'Executable doesn't exist'. Fetch the launched browser (with system deps) after yarn install. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ope nyc to the core surface nyc instruments the arrow functions dropin/dom.js ships to the browser via locator.evaluate, and the injected coverage counters don't exist in the page (ReferenceError: cov_… is not defined) — annotate them with istanbul ignore, the same convention index.js/utils.js already use for browser-executed code. Exclude dropin/ and bin/ from the 100% coverage gate for now: the modules are covered by the standalone package's 154-test unit suite, and porting that suite into this harness is a flagged fast-follow. The existing core surface keeps its 100% bar unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Which fallback executes depends on the consumer's installed packages — the defensive branches can't all be exercised in one environment and were the last gap under the 100% coverage gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Node 18 went EOL in April 2025; its leg also mis-collects subprocess coverage for utils.js under the percy-exec harness (identical 117-test pass count, different coverage) — not worth chasing on a dead runtime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolves the 7 blocking semgrep findings (path-join-resolve-traversal, CWE-22): NUL bytes are stripped from any externally supplied directory before it reaches path.join, and directory-walk entry names must be a single path component (no separators, no dot/dot-dot) or the entry is skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports the
toHaveScreenshotdrop-in (PER-8985) into this SDK so it ships from@percy/playwrightinstead of a separate package. One require line converts a repo's existing Playwright screenshot assertions into Percy visual tests — no test changes, no rewrites.What's in the box (
dropin/)dropin/index.js): replaces Playwright's built-intoHaveScreenshotmatcher on the sharedexpect. The publicexpect.extend()silently ignores built-in matcher names on the shared instance (the override only exists on the discarded return value), so registration writes into expect's internaluserMatchers— which call-time dispatch spreads after built-ins. On an unrecognized internal shape it degrades toextend()with a loud warning, never a silent no-op. Verified on@playwright/test1.60/1.61.dropin/identity.js): snapshot names reproduce Playwright's on-disk baseline stems byte-for-byte (same sanitizer regex, same per-test anonymous counters) so uploads pair with committed baselines..percy-playwright-dropin.json):screenshot(default): stabilized PNG → raw-image comparison (generic/app projects); tag width = identity width, tag height parsed from PNG bytes (percy-api validates height presence).snapshot: serialized-DOM web snapshot for web projects — delegates to this repo's owncaptureDOM, inheriting the readiness gate, responsive capture, and cross-origin iframe handling; the two entry points cannot drift. Locator subjects become element-scoped snapshots (transientdata-percy-dropin-scopemarker +scope).percy-playwrightwrapper bin setsPERCY_DROPIN_BASELINE_CANDIDATE=true(mirrorsPERCY_BUILD_SOURCE); percy-api decides first-ness server-side and, on a genuine first build,dropin/global-setup.jsseeds the repo's committed baseline PNGs as the build's content (8-way parallel) while the override skips live captures. The server auto-approves that baseline (org-flag-gated) — diffs start on run 2. Server side lands via percy/percy-api#6178; CLI plumbing via feat(client/sdk-utils): additive support for @percy/playwright-dropin baseline seed + sync verdict cli#2275.dropin/reporter.js(fail-on-changes CI gate, first build always review-only) and sync-assertion mode (inline verdicts,.percy.yml snapshot.sync, read-token guarded).Repo-level changes
index.js: exportscaptureDOMfor internal reuse;ENV_INFO's top-levelrequire('playwright/package.json')crashed at import time under the 1.61 runner (CJS↔ESM interop) — now guarded with graceful fallbacks.package.json: shipsdropin/+bin/, adds thepercy-playwrightbin,@playwright/testas an optional peer (only the dropin entry needs it), devDeps bumped^1.24.2 → ^1.60.0(the drop-in requires modern expect internals; existing suite passes unchanged on 1.61).Testing
@playwright/test1.61.1), lint clean.expectthrough the override and hard-assert the posted comparison (name/tag dims/tile) — the exact check that prevents a silent registration no-op; wrapped inexpect().toPass()because the shared CLI testing server is reset concurrently by parallel workers.Notes for reviewers
META_INFOsymbol /userMatchers); the degrade path is loud by design. This is the part most worth scrutiny.dropin/is a direct port of percy/percy-playwright-dropin PR #2 minus its standalone DOM capture (replaced bycaptureDOMreuse). If this lands, the standalone package can be retired before ever publishing.🤖 Generated with Claude Code